Out-of-core minibatch ADVI on a financial tick stream - #892

Update the out-of-core variational inference example to the reworked
streaming API: a DataLoader over parquet_source, a pm.Data placeholder, and
the callback-free Trainer (replacing the removed StreamingDataset and
fit_callback). Replace the non-reproducible private-data memory note with
measured peak-RSS figures on the public Criteo benchmark. The notebook
executes end-to-end; outputs and figures regenerated.

Set layout=tight at figure creation instead of calling tight_layout on
a constrained-layout figure, which emitted a warning into the committed
outputs. Declare pyarrow via extra_dependencies and the standard
install note. State that the Criteo numbers come from a run outside the
notebook, fix the out-of-memory extrapolation to follow from the stated
measurements, and correct two sentences that overstated what stays in
memory.

- pass random_seed to the in-RAM pm.fit and to both posterior sample
  calls so the comparison is reproducible
- the streaming memory line now includes the shuffle buffer and the
  current source chunk, matching the surrounding prose
- say 'no callbacks to write' (the Trainer uses pm.fit's hook
  internally) and 'can bias' for the bounded-buffer caveat
- note that the dropped end-of-pass remainder is re-drawn each epoch
  under shuffle=True

The largest streaming vs in-RAM posterior gap was 0.12 on a feature
coefficient, not ~0.1 on the intercept (the intercept gap was 0.01),
and the comparison ran on a 1M-row slice; the OOM extrapolation from
the fitted slope is ~240M rows, not 250M.

The artifact shows two near-zero coefficients flipping sign between the
two stochastic fits, so 'agreed coefficient for coefficient' overstated
it; state the max gap against the coefficient scale and disclose the
sign flips. parquet_source reads row groups, not whole shards.

Plainer section headers (Write the dataset to disk; Compare with in-RAM
pm.Minibatch), walk-through connectors, and a tighter stream-and-fit
lead-in that no longer repeats the intro. Markdown only; outputs
unchanged.

On the 1M-row slice two near-zero slopes flip sign and the streaming
estimates sit ~5 sd from zero, so 'two near-zero coefficients differed
in sign' understated it; say plainly that the weak slopes disagree. The
memory paragraph now notes the figure ignores the np.concatenate copy,
and the comparison paragraph states the fits share everything but the
minibatch source.

Applied capstone for the streaming stack: hierarchical hurdle-Student-t
next-event returns on the Binance public archive, with the executed path
on synthetic data and full-scale results (491.6M rows, 38 symbols)
inlined from published run artifacts. Companion to streaming_dataset.

…ated claims

…k pair

… scoped claims

…ding fixes

…c labels

The merged loader deleted sample_shape, made len() the batch count with
.total_size carrying N, and made shuffle=False a verbatim block
pass-through. Shards are now written with row_group_size=1024 so the
pass-through yields the intended minibatches (288 full blocks plus 8
short ones of 636 rows per epoch, 300,000 rows exactly). The drop-last
tempering paragraph is gone with the behavior it described: nothing is
dropped on this path, and the total_size rescaling reads the installed
batch's own size, verified mechanically (model logp identical for
b=10/4/7 under total_size=100).

The stopping-rule demonstration reproduces under the new batching: the
raw per-step rule still false-stops ~80 steps after arming (371), the
windowed rule stops at 4,650 of 20,000. Its intro now notes that the
trap shown here was later confirmed on real ADVI traces and that the
revised implementation under review for pymc-extras builds on the same
windowed idea.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

# Conflicts:
#	examples/references.bib

…ymc-devs#883

The bib rebuild merged in pymc-devs#883 lost this entry while fixing the earlier
duplicate-key corruption; this notebook and the SVI literature trail cite
it. Verified by key-set diff against the pre-merge revision: it was the
only entry lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

streaming_dataset.{ipynb,myst.md} predate the merged DataLoader: they import
a pymc.variational.streaming module that does not exist and present a
Trainer as available. The proposal in pymc-devs#891 commits to one example, not
two; this PR now contains only the tick-data notebook and the bib fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three independent audits of the previous revision found nine blocking
claims, most of them inherited from the first draft and never checked
against the executed outputs. The predictive prose said two of three
statistics were inside their bands when all three are; the estimand
paragraph promised a demonstration that never came and that the
generator contradicts (nu = 3.5, so the moment-based dispersion exists);
`len(loader)` had two incompatible definitions eight hundred lines
apart; the covariate paragraph described causal feature engineering,
a log notional and per-symbol standardisation, none of which the
generator does; and the hierarchy figure claimed shrinkage that it
cannot demonstrate without an unpooled comparison it never computes.
All are corrected or withdrawn.

Structural changes in the same pass. The epoch boundary is now measured
(296 blocks, 300,000 rows conserved) rather than taken from
`total_size // batch_size`, which is four short on the verbatim path.
The falsified per-step CUSUM demonstration is gone along with its
second 20,000-step fit; in its place a signed block-contrast figure on
the existing trace shows what a stopping rule can see at three
horizons, with a 1.5-epoch control that isolates the mechanism: the
variance collapse at epoch-aligned windows is twenty times smaller
than at the misaligned one, because a fixed replay order averages the
same rows each pass. A retrospective t99 benchmark replaces the claim
that a rule "stops after convergence" (and its index into the
convolved array is now mapped back to stage-2 coordinates).

Additions: the event clock is defined before the algebra, so pi and
sigma are not read as clock-time quantities; every generator constant
is annotated as the setting it is; a new figure translates the
posterior into the two curves the model exists to produce, medians and
truth only, no epistemic ribbons; and a closing component map names
all four pull requests with their form here. Nothing unmerged is
imported or copied: the convergence callback and the streaming
Pathfinder appear as the problem they solve and as an explicit
pre-fit applicability decision.

Cold-kernel runs: 25.2 / 21.5 / 21.8 s, peak RSS 872 MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The citation left with the executable CUSUM copy, but the mechanism
paragraph that replaced it still describes the one-sided cumulative sum.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

gelman2006data and hoffman2013stochastic carried neither doi nor url, so
sphinxcontrib-bibtex rendered them as plain text while every other entry
on the page links out. Both verified against CrossRef metadata (book DOI
10.1017/CBO9780511790942) and the publisher's own listing (JMLR v14).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An external review raised two blockers; both reproduce.

First, `approx.hist` is not the full-data negative ELBO the notebook said
it was. PyMC normalizes twice: MinibatchRV scales the observed logp up by
N/b, then OPVI divides the objective by the same constant because
`scale_cost_to_minibatch` defaults on. What is recorded is the batch's own
log-likelihood sum plus (b/N) KL, which scales linearly with b — measured
directly by alternating 1000- and 400-row blocks and reading the ratio of
mean recorded loss: 0.4126 against the 0.400 the batch sizes predict.
Multiplying by N/b restores the full-data scale (0.9998 against a full-data
reference over 1200 steps). The notebook now derives this, plots and
analyses `hist * N/b`, and reports 188 nats of stage-2 reduction rather
than the 0.60 that the batch scale produced.

Second, the shard geometry made blocks ragged: 288 of 1024 rows plus eight
of 636, which put a deterministic sawtooth into the loss and left
`len(loader)` matching neither the block count nor the yields. Ten shards
of 30,000 rows with row groups equal to the batch size give exactly 300
equal blocks, `len(loader) == 300`, and a loss series whose only structure
is optimization and noise. The per-step standardized contrast now has unit
spread (0.99, from 1.07) and the epoch-alignment result survives on the
clean series: the spread at epoch-aligned horizons is more than twenty
times smaller than at 1.5 epochs.

Third, provenance. The stored outputs were produced in an environment
whose pymc-extras metadata predated the merge and whose PyMC violated the
merged package's own floor. Everything here was re-executed against
pymc-extras at 8db1880 (the pymc-devs#698 merge commit), PyMC 6.2.0, PyTensor
3.2.4, and the install note now gives readers that exact command.

Also corrected: the CUSUM warning overstated its case (the recursion sits
at zero while improvement exceeds the allowance; the real failure is that
it cannot tell convergence from improvement too slow to resolve); the
fixed replay is described as cyclic finite-sum optimization rather than
the unbiased N/B identity; the hurdle alone does not prevent sufficient-
statistic collapse; the nu floor is a statement about support, not prior
belief (P(nu<=2) is 4e-6); the row-scale generator arrays are released
before fitting and the in-memory preprocessing is labelled as such; the
shrinkage claim in the model section is withdrawn; and the predictive
check is described as one batch, one step ahead.

Cold runs in the pinned environment: 11.0 / 10.0 / 9.6 s, peak RSS 619 MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The generated dependency line asked for a released pymc-extras, which does
not contain the DataLoader, while the note underneath said to install from
an unpinned main. Neither reproduces the environment in the watermark.
The dependency line now covers only pyarrow and graphviz, and the note
carries the pinned commit on its own.

The substitution lives in notebook metadata as well as in the front
matter, and the jupytext hook rewrites the text file from the notebook, so
both copies are updated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two defects, both invisible from the source and both visible on the built
page.

The front matter declared pyarrow as an extra dependency, but the notebook
never included ../extra_installs.md, which is the only thing that consumes
that substitution (conf.py maps pip_dependencies -> extra_dependencies).
Nineteen notebooks in the repo declare extras; seventeen include the
snippet. This was one of the two that did not, so the declaration rendered
nowhere and a reader following the page installed pymc-extras and then
failed on `import pyarrow`. graphviz is dropped from the declaration
rather than added to it: fifty-five notebooks call model_to_graphviz and
none of them declare it, so the repo treats it as assumed.

The recovery section then argued from a number it refused to print.
"Notice also what we did not print: z-scores against the truth. Some would
exceed 2" understates by a factor of three, and the reader can compute the
column from the three columns that were printed. It is now the fourth
column, and the prose reads it against identification instead of asserting
a bound on it: the raw split rows are not recovery statistics at all
(alpha0 reads 404, which is a statement about an arbitrary decomposition),
the identified sums are all inside 2, and between them sit theta_d at -7.1
and lambda_q at +4.9 -- cleanly identified rows that miss by more than
their own width allows. The withheld-width stance survives, but it now
rests on a printed number rather than on a characterization of one.

Re-executed in the pinned environment; every other output is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An external audit raised the last-iterate question indirectly: stop on an
epoch boundary, it said, or show that epoch-boundary checkpoints are
stable. Stopping on a boundary (2,400 steps = 8 passes) moved the recovery
table enough to be alarming -- theta_d went from -7.1 to +2.8 posterior sd
on 100 fewer steps -- so the question was worth measuring rather than
assuming.

Two measurements settle it. Between the same phase of consecutive epochs
the variational mean moves a median of 0.11 and at most 0.58 posterior sd:
the descent really has flattened. Within a single epoch it swings a median
of 4.3 and a maximum of 41.2. The batch at step t is a deterministic
function of t with period one epoch, so the gradient sequence is periodic
and the iterate orbits the optimum instead of settling on it. The last
iterate is whichever phase of that orbit the step budget ended on, which
is exactly what the 2,400-vs-2,500 difference was.

The fix is the standard one: average the variational parameters over the
final whole pass, a full period, which is what makes the average cancel
the cycle. Every identified quantity then lands inside 1.2 posterior sd of
its generating value -- theta_d at -0.05, theta_r at +0.02, the three
identified sums at 0.14, 0.22, 0.23 -- against misses of three to seven
before averaging. The ridge splits are unchanged, as they must be.

This also closes the audit's other reading of the recovery section, that
one fit cannot say whether a gap is a too-narrow width or a biased mean.
Most of the gap was neither.

Also from the audit: Gate 1 now separates the factorization requirement
from how batches are drawn, instead of claiming the notebook's cyclic
replay satisfies a uniform-sampling condition; the disk shuffle is
described as a fixed hash ordering rather than a uniform permutation drawn
at random; the shard-head check asserts over all ten shards instead of
printing one; the memory-release cell releases the four row-scale arrays
it had left live; the plate diagram is described as an inventory rather
than as showing two predictor links it does not contain; the ADVI citation
moves off PyMC's total_size semantics, which it does not discuss; the
Fourier citation is narrowed to what Andersen and Bollerslev support; the
w^-3/2 claim states its noise assumption; the ragged-batch sawtooth is no
longer cross-referenced to a section that never demonstrates it; the
control/aligned spread ratio is printed rather than asserted; the
predictive check reports its margin on the zero share; and pymc.Minibatch
uses the func role its inventory declares.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A second audit round, this one aimed at the tail-averaging section, found
six things; five held.

The largest changes what the recovery section says about kappa0, alpha0
and beta_a. The notebook called them "prior-dependent decompositions, not
estimates of their generating values" and said "no amount of data fixes a
ridge". The likelihood is exactly flat along (alpha0 + d, b - d), but the
hierarchical prior tilts that direction toward effects that average to
zero -- and the generator standardized them to average exactly zero -- so
the prior points the split at the generating value with a posterior width
near tau/sqrt(12). Along a direction that flat a gradient optimizer
crawls. A new cell warm-starts a fresh optimizer at the last iterate and
runs sixty more passes: kappa0 goes 0.532 -> 0.579 -> 0.628 toward 0.847,
alpha0 -1.702 -> -1.722 toward -2.996, while the loss falls 0.37 nats per
pass. The raw split rows are a fit still travelling along a nearly flat
direction, and their reported widths (a few thousandths) are the
mean-field conditional width across the ridge, not the tenth the ridge
posterior has along it. Both are now said; both are reasons for the
sum-to-zero reparameterization the notebook already recommends.

Second, "the mean barely moves, so the descent really has flattened"
rested on the last of seven same-phase increments, which happened to be
the smallest. The cell now prints all seven maxima (6.37 2.93 2.59 0.75
1.39 1.45 0.58) and the last-pair median (0.11), and the prose says what
that supports: the phase dominates the last iterate, the average removes
the phase, and a slow drift remains that the average does not remove.

Third, two counts were wrong: "four of those nine are inside 0.5" is
seven; "misses of three to seven" without the tail average is a range no
row occupies -- the last iterate runs to 3.3. The replicate cell now
computes the last-iterate summary so that number is stored, and adds two
independent-seed refits: max |z| on identified rows 1.16 / 1.15 / 1.14,
across-seed spread of the means about a tenth of the reported width. The
"replication can veto a width" paragraph now describes a check that was
run.

Fourth, "the visible band narrows after the change" is false (sd 13192
before the anneal, 13141 after); the band is batch-composition noise and
the learning rate does not touch it. The sentence now says where to look
instead. The loss figure's two-line title, which overran the axes, is one
line.

Fifth, the warning filter's comment described a RuntimeWarning that never
fires; only the two numba object-mode warnings do.

One finding was refuted: the pushed head of pymc-extras#722 (f98a4ff)
does Polyak-Ruppert tail averaging over 75% of the trajectory, so the
notebook's statement about it stands; the reader had a stale local branch.

Also: the alignment claim now says averaging over a whole number of
periods removes the phase dependence exactly and a misaligned window
leaves a residual of order one swing over the window length; the
duplicated Gate 1 paragraph is cut to what it adds. Cold runs 29.1 / 31.2 s
(three fits plus the continuation), peak RSS 587 MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…node (429)

freezing review state into the page

Third audit round; eleven of fourteen findings held.

The replicate cell divided every fit's error by seed 0's posterior sd,
while the intro said "its own posterior standard deviations". summarize()
now returns mean and sd for every reported row and each fit is judged in
its own width: last iterate 3.42, seed 0 1.16, seed 1 (same order) 1.14,
seed 2 (other order) 1.13. The second refit runs on a different on-disk
order -- the rows re-keyed with a different hash salt and re-sharded --
because the replay order is the one ingredient the orbit account is
about, so it is the one worth varying rather than holding fixed. Spread
of the mean across the three tail-averaged fits: median 0.08, max 0.10
of the reported width.

"Every identified quantity" now says what it means: every identified row
the recovery table reports, nine of them. The claim about the ridge width
is stated as the conditional calculation it is (Gaussian in the shift
with sd tau/sqrt(12), a tenth for tau near the generator's 0.35), not as
a measured marginal. The orbit account no longer says the gradient
sequence is periodic -- the data term is; the parameters and the
Monte-Carlo draw are not -- and no longer says one pass removes the
phase "exactly". The pymc-devs#722 sentence says "the same device for a
neighbouring reason" instead of "for the same reason". "Cancels" at
aligned horizons becomes "all but drops out", which is what sd 0.02-0.03
means. The loss cell prints the recorded sd on both sides of the anneal
(13,192 vs 13,141) so the sentence about the band rests on an output.

The continuation labelled its first recorded point +0 when the callback
fires after the first epoch; the pre-continuation parameters are now
recorded as +0. The post date carries a day, as the siblings' do, so
ABlog stops filling it from the build clock. Two DataLoader roles that
cannot resolve (the pmx inventory does not have the class and conf.py's
mapping points at a moved URL) are plain links to the pinned source. The
prose no longer says pymc-devs#710 "proposes", pymc-devs#635 "develops", or pymc-devs#733 is "under
review" -- state verbs that go stale; it says what each PR contains. The
out-of-core paragraph says a hash-assigned two-pass design yields an
equivalent shuffle, not the same permutation. The notebook times itself
and the watermark names the machine (a second %watermark call, so the
repo hook's regex still matches).

Not changed: the pixi environment (a repo decision, raised in the PR),
and the framing (author's call).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A last review before this goes to the maintainer, aimed at two things:
whether the prose reads as written by a person, and whether the figures
show what the text says they show.

Prose. Twenty-four passages that announced their own importance ("worth
stating plainly", "That is not a subtlety to note and move past", "Read
the two measurements against each other", "Two lessons sit on top of each
other here") or labelled their own virtue ("Honest reporting", "honesty of
the showcase", "genuinely disk-backed") are rewritten to say the thing
without the frame. Em-dashes go from 73 to 2 (the two Gate title
separators); asides become parentheses, commas or their own sentence. The
paragraph-closing aphorism about replication is cut; the sentence before
it already says what it says. No number, citation or caveat changes.

Figures. The loss plot clipped y to the 0.2-99.5% quantiles of the whole
trace, which the first few hundred steps stretch to +645k, so the plateau
was an unreadable ribbon and the title's "clipped to the plateau" was
false. It now clips on the loss after step 1,000, overlays a one-epoch
moving mean, and prints the spread and level over the two passes either
side of the learning-rate cut (sd 13,192 vs 13,093; mean -96,528 vs
-97,746) so the prose about the cut rests on an output. The block-contrast
plot's aligned horizons were invisible at the per-step scale, its title
overran the y-label, and its frameless legend sat on the spikes; it is now
two panels, the second zoomed to +/-0.12 where the drift the prose
describes can be seen. The predictive ECDF's legend moves off the curve.

Page. The component map is retitled "Related tooling in pymc-extras",
loses its self-review column and status framing, and absorbs the
Pathfinder applicability note; two inline PR mentions go, since the map is
their home. Long print statements are split so no number is cut off at
the output cell's width; the temp path is reduced to its basename; the
install line uses the shorter git+ form so the hash is not clipped; the
watermark no longer lists pytensor and pymc_extras twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ings

Checked against the PyMC Jupyter style guide and the five notebooks in
the same section. Mechanics already matched (front matter, first cell,
extra_installs include, seed/rng, arviz-variat, citations, end-matter
order, watermark hook). What did not:

- "RVs" -> "random variables" (the guide's own example of an abbreviation
  to avoid); ETL, SGD, VI, ADVI, ELBO, MCMC spelled out at first use;
  "pp mean" / "lr" / "mean-field sd" labels expanded; ECDF expanded in
  the prose before the figure; `icpt` -> `intercept`.
- The batch size was written $B$ in the gates and $b$ in the objective,
  and $B$ also named the hour basis. It is $b$ throughout the prose now
  and the batch index set is $\mathcal{B}_t$; the symbols in the recorded
  objective are defined under the display, and the model equation gets
  a "where" list naming every coefficient, per the guide.
- `axes` -> `axs`; `plt.matplotlib.ticker` and the blocking bijection are
  imported at the top; `for a in (ax, ax_zoom)` no longer shadows the
  covariate `a`.
- British spellings (artefact, generalisation, standardised, favour,
  initialise, neighbours, travelling, Monte-Carlo) normalized to the
  American forms every sibling uses.
- "Notebooks in this collection do not download data at build time" was
  false (37 of them do); it now says only what this notebook does.
- Polyak-Ruppert averaging is cited (Polyak and Juditsky 1992, verified
  against CrossRef); the entry is added to references.bib in order.
- The GSoC credit moves out of the Authors bullet into an
  Acknowledgements section (the MOGP-Coregion-Hadamard precedent), and
  the Authors line follows the guide's pattern with the PR link.
- Watermark adds `-p xarray`, which is used without an import.

Not changed, on purpose: the impersonal voice (the siblings say "we"; the
guide is silent, and switching mid-notebook would be worse than either),
and the hand-rolled figures (each replaces an ArviZ function for a stated
reason in its cell comment).

Re-executed in the pinned environment; the machine was under load from
unrelated jobs, so the self-reported wall time reads 42 s against 28 s
idle.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>